-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor CloudWatch related code and add logs input #43
Conversation
bernd
commented
Aug 10, 2017
- Refactor CloudWatch related code
- Create a generic Kinesis transport and reader that can be used by different inputs
- Create an abstract CloudWatch log data codec
- Rename some flow log classes to more generic names
- Update to the latest AWS SDK and Kinesis client
- Use unique application name per stream to avoid state problems in DynamoDB
- Periodically checkpoint the sequence number to make sure the consumer continues where it left off instead of only fetching new data after a Graylog server restart
- Use auth provider in InstanceLookupTable so it also works with instance profiles, environment variables, etc
- Inject InstanceLookupTable and bind it as singleton
- Create a generic Kinesis transport and reader that can be used by different inputs - Create an abstract CloudWatch log data codec - Rename some flow log classes to more generic names
The Kinesis client is using a DynamoDB table under the hood to track shard offsets. When a consumer is using the same application name for two different Kinesis streams, checkpointing will fail.
This ensures that we are getting old records after a Graylog server restart instead of starting with LATEST on each restart.
Also use AmazonEC2Client builder instead of deprecated constructor.
- Remove now unused "AWSPluginConfiguration#isComplete()" - Log a warning if the lookup processor is enabled but no regions are configured
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @bernd! LGTM (Left one tiny comment)
@@ -12,14 +14,14 @@ | |||
import javax.inject.Inject; | |||
|
|||
public class FlowLogsInput extends MessageInput { | |||
private static final String NAME = "AWS FlowLogs Input"; | |||
private static final String NAME = "AWS CloudWatch Flow Logs"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tiny minor thing: I'd rename this to just "AWS Flow Logs", because CloudWatch is really just a transport and it could be a little confusing for users who think that this is Flow Logs for CloudWatch.